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Abstract. We investigate the structure of invariant distributions on a non- 
isotropic non-Riemannian symmetric space of rank one. Especially, the J- 
criterion related to the generalized Gelfand pair is shown for this space with- 
out imposing the condition on the eigenfuction of the Laplace-Bertrami op- 
erator. 

1 Introduction 

In this paper, the problem [TT] to understand the double coset space H\G/H 
in terms of functions of distribution class for a homogeneous space G/H is ex- 
amined in one specific case study for the space G/H = SL(n+l, R) / GL + (n, R). 
If one deals with the class of arbitrary functions, then the H x if -orbit decom- 
position on G gives enough information. This is the case for the symmetric 
spaces over the finite field (c.f. [15] ) and the set of characteristic functions of 
orbits forms a linear basis of invariant functions. If one deals with the regular 
functions on the algebraic group, it is enough to look at the ring of invari- 
ants. More generally, if one considers bi if -invariant continuous functions, 
then the lower-dimensional orbits can be ignored. 

We here consider distributions. The group case, the famous result of 
Harish-Chandra tells us that every invariant eigendistribution is locally in- 
tegrable and that there is no singular invariant eigendistribution. For Rie- 
mannian symmetric spaces, every invariant eigendistribution is real analytic. 
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However, for the case of non-Riemannian symmetric spaces, the natural func- 
tion space for invariant eigendistributions (spherical functions) is the class of 
distributions (or its generalization to hyperfunctions). A lot of works have 
been done for spherical functions and also the eigenspace representations, 
which imposes the eigenspace condition without invariance. Here we will 
consider invariant distributions without the eigenfunction condition of (in- 
definite) Laplacian. In the case of invariant eigendistributions, the space is 
finite-dimensional, and especially for the rank-one case we can write down 
explicitly such distributions using e.g., hypergeometric functions. On the 
other hand, the space of invariant distributions is infinite-dimensional, and 
has no ring structure in general. 

Our target space G/H is a symmetric space, however, we use the related 
homogeneous spaces, having a larger symmetry. In the case of the analysis on 
the tangent space of the symmetric space, such an enlargement of the sym- 
metry, which has been discussed in jH] (see also |Zj), can be formulated on the 
tangent space itself, but on the symmetric spaces it is necessary to introduce 
such homogeneous spaces to formulate the enlargement of the symmetry. 

Anyway, for the symmetric space or its related (spherical) homogeneous 
space, the double coset space H\G/H is not a manifold especially near the 
origin. It is neither smooth nor Hausdorff, in general. Since we are consider- 
ing distributions, the analysis at the origin is complicated on such a space. 
For a non-archimedian (p-adic) local field, the structure of distributions are 
easier than for real numbers R, e.g., the extensions and decomposition of 
distributions reduces some problem to the orbit decompositions, and the 
structure of the distributions supported on a lower-dimensional submanifold 
is simpler (there is no derivative of Dirac delta), c.f. pQ. In our Lie group 
case, analysis around some singular locus will be more subtle, and it is a 
point of discussion. We apply the result [7] on the tangent space of G/H to 
prove the extra symmetry which every invariant distribution has. This en- 
ables us to eliminate a contribution of some singular loci on the non-isotropic 
symmetric spaces. This is given in Section H3 In Section El we summarize 
several geometric facts, which will be used in the later sections or help us to 
understand the double coset space. 

As an application of the main theorem in Section we discuss the prop- 
erty called generalized Gelfand pair in Section |3J This notion is related to 
the uniqueness of the decomposition of the left regular representations of 
L 2 (G/H), see [TJ]. So-called J-criterion is one of the well-known sufficient 
conditions to be a generalized Gelfand pair. The J-criterion for a class of 
semisimple symmetric spaces of rank one is examined in • Here we slightly 
generalize the statement by dropping the eigenfunction condition. Note that 
the J-criterion holds while the geometric counterpart does not hold. It sug- 
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gests a careful study between the orbit spaces and the space of invariant 
distributions. 

The author would like to his gratitude to Professor dr. van Dijk, and 
organizing committee of the conference. 

2 Geometric background 

2.1 Enlargement of the symmetry of the space 

Let G = SX(n+l, R), and a be the involution of G defined by the conjugation 
by the matrix diag(— 1, !,...,!). The fixed point subgroup is 



which is often to be identified with GL(n, R). The identity component of 
G a is denoted by H, which is isomorphic to GL + (n, R). The homogeneous 
space G/H is a non-isotropic non-Riemannian symmetric space of rank one, 
which is our main concern in this paper. We introduce H\ = SL(n, R) C 
H = GL + (n, R), then we have G a /H = Z/2Z, G a / H x = GX(1,R) = R x , 
H/Hx = GL+(1,R), and H = K x x GL+(1,R). Then we have G/H = 
(G/Hi)/(H/Hi) = (G/H 1 )/GL + (l,K). It is easy to see that the normalizer 
of Hx in G is G u . 

We will describe the homogeneous space GjH\ as follows. Let X\ = 
{(x, y) | x, y G R™ +1 , (x, y) = 1}. Here we regard x = t (x±, X2, ■ ■ ■ , x n+ \) = 
y = t (yi,y2, ■ ■ ■ ,y n +i) — * (2/1 ? 2/') as column vectors. The action of 
G on X\ is defined by 



It is easy to see that this action is transitive. We set x\ = t ( t ei, t ei) with 
e\ = 0, . . . , 0) G R n+1 . Then the isotropy subgoup of G at X\ is Hi, and 
we have a natural isomorphism X\ = G/ Hi . 

We define the function Qo(x,y) = Xiyi + x 2 y2 + - • ■ + x n+ iy n+ i on R 2 ^ n+1 ). 
We denote by G = SO (Q ) = SO (n + 1, n + 1) the identity component of 
the orthogonal group O(Q ) corresponding to the quadratic form Q . Then 
G also acts on Xi transitively. The isotropy subgroup H 2 of G at x\ is 
isomorphic to SOo(n,n + 1). Then X\ = G/Hi = GjH 2 - The role of this 
isomorphism for the harmonic analysis has been emphasized in [3] , Example 
5.2. This isomorphism means that the space G/Hi has a larger symmetry 
G. The expression Xi = G/Hi is not a symmetric space but a (real form 
of) spherical homogeneous space. The expression Xi = G/H2 is an isotropic 
symmetric space of rank one. 




9-(x,y) = (gx, l g 1 y), 



x, y e R'' 



n+l 



gEG. 
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2.2 Invariants and orbits 

The function Q(x, y) — 1 — xiyi = t x'y' on X\ is if-invariant. The functions 
X\ and yi on Xi are f/i-invariant. The map 

q: X x 3 (x,y) h-> (xi,yi) G R 2 

is real-analytic, surjective, and if i-invariant. As is seen later, the map q 
almost classifies the set of if i-orbits on Xi, and any ifi-invariant continuous 
function on Xi is a pull back by the map q of a continuous function on R 2 . 
For distributions, the question is more subtle. 

Let H = SOq(Qi) be the identity component of the orthogonal group 
0(Qi) corresponding to the quadratic form Qi(x',y') = x 2 y 2 + - • • + x n +iy n +i 
on R 2n . Then H is identified with a subgroup of G — SO (Q ). The group H 
preserves the invariants x\ and y\. Now we give several orbit decompositions. 

(1) iii-orbit decomposition on X\. The map q = (xi,yi) almost classifies 
the iii-orbits on X 1 . Say, for (xi,yi) G R 2 with x±yi ^ 1, the fiber q~ 1 (x\,yi) 
is an i?i-orbit if n > 2. For x\ ^ 0, the fiber q~ 1 (x±, x^ 1 ) splits into four H±- 
orbits {(x',y') | tx'y' = 0,x'^ 0,y' ^ 0}, {(0,y') | y' ^ 0}, {(x',0) | x' ^ 0}, 
and {(0,0)} if n > 3. If n — 2, then the fiber q _1 (xi, x 1 1 ) consists of Hy- 
orbits {(x',y') = (x 2 , x 3 , y 2 , y 3 ) \ x' ^ 0,y 2 = -tx 3 ,y 3 = tx 2 } with t ^ 0, 
{(0,y') | y' ^ 0}, {(x',0) \ x' + 0}, and {(0,0)}. 

(2) GL + (n, R)-orbit decomposition on X±. We consider the action of 
GL + (n, R) on X x by 

(xi, x', yi, y') h-> (xi, for', yi, */tY) for ft, G GL + (n, R), (x, y) G Xi. 

The orbit decomposition on X 1 under GL + (n, R) is the same as that under 
Hi, except for the case n = 2 and the fiber q~ 1 (xi, xf 1 ), which is decomposed 
into five GL + (n, R)-orbits {(x', y') = (x 2 , x 3 , y 2 , y 3 ) | x' ^ 0, ±(x 2 y 3 -x 3 y2) > 
0}, {(0,/) | y' ^ 0}, {(x', 0) | i7 0}, and {(0, 0)}. 

(3) if -orb it decomposition on X\. Since the map is if -invariant, the fiber 
q~ 1 (xi,y±) for (xi,yi) G R 2 with xiyi ^ 1 is an if-orbit if n > 2. The fiber 
q~ 1 (xi,Xi 1 ) consists of two if -orbits, {0} and {(x',y') ^ | t x'y' = 0}. 

(4) if-orbit decomposition on X. This is equivalent to GL + (n, R) x 
GL + (1, R)-orbits on X\. In this case, the map Q : Xi 3 (x, y) \— > 1 — Xiyi G 
R almost classifies orbits. In fact, for t ^ 0, 1, the fiber Q _1 (t) consists of 
two orbits {(x, y) G Xi | Xiyi = 1 — t, ±xi > 0} if n > 2. Q _1 (l) consists of 
five orbits {(x, y) G Xi | ±xi > 0,yi = 0}, {(x, y) G Xi | Xi = 0, ±yi > 0}, 
and {(x,y) G Xi | Xi = y x = 0} if n > 2. For xi ^ 0, the fiber Q _1 (0) 
splits into eight orbits {(xi, x', xf 1 , y') | ±xi > 0, t x'y' = 0,x' ^ 0,y' ^ 0}, 
{(xiAx^y') | ±X! > 0,y' ^ 0}, {(x 1; x', xf\ 0) | ±Xx > 0,x' ^ 0}, and 
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{(xi, 0, x 1 , 0) | ±xi > 0} if n > 3. If n = 2, then the fiber q 1 (0) consists 
of ten orbits {(aci, x 2 , x 3 , x]" 1 , y 2 , Z/3) I ±a?i > 0, a;' 7^ 0, ±(x 2 l/a - 2*2/2) > 0}, 
{(xi,0,Xi ,y') I ±xi > 0,y' 7^ 0}, {(xi, x', x± , 0) | ±xi > 0,x' 7^ 0}, and 
{(xi,0, av\°) I ±xi > 0}. 

(5) H x GL + (1, R)-orbit decomposition on X\. For n > 2 and £ 7^ 0, the 
orbit decomposition of the fiber is the same as that of (4). The fiber 

Q _1 (0) splits into four orbits {(xi, x',Xi ,y') \ ±Xi > O^x'y' = 0, (x',y') 7^ 
0}, and {(xi,0,x^,0) | ±x x > 0} if n > 3. If n = 2, then the fiber Q _1 (0) 
consists of four orbits {(xi, x%, X3, x^ 1 , 2/2, 2/3) | ±^1 > 0, (V, j/) 7^ 0}, and 
{(xi,0,£r\0) I ±xi > 0}. 

The geometric J-criterion (for an involution 6) is the statement that 
Hg~ x H = H6(g)H for all g E G. Note that for the involution 0(g) = t g" 1 , 
the space G/H does not satisfy the geometric J-criterion since the if -orbit 
{(xi, 0, x± , y') I ±xi > 0, y' 0} is mapped to {(xi, x', , 0) | ±xi > 
0,x' 7^ 0}, respectively, by HgH 1— > H9{g)^ 1 H . Nevertheless, we will prove 
the (original) J-criterion (for distributions) in Section 0] 

3 Bi-invariant distributions 

We denote by C~°° the set of functions of distribution class. We simply call 
a function of distribution class a distribution. 

We start from a direct conclusion of the enlargement of the symmetry of 
the space Gj ' H\. 

Lemma 1 There are natural identifications between the set of distributions 
with the following properties. 

(i) An H -bi-invariant distribution on G. 

(ii) A left H and right H / H\-invariant distribution on G/H\. 
(Hi) A left GL + (n,H) x GL + (1,R) -invariant distribution on X x . 

Proof. Since H normalizes Hi , H acts on GjH\ from the right as 

G/H x 3 gH x 1- ghH x G G/H u 

which induces the action of H/ Hi = GL + (1, R) on X± as (x, y) i— > ((det /i) _1 x, (det h)y), 
that is, (x,y) 1— > (t~ l x,ty) by t > 0. On the other hand, the action of H on 
G/ Hi from the left is 

(xi, x', yi, y') h-> ((det h)~ 1 xi, hx', (det h)yi, */i~V)- 



5 



Then, an if-bi-invariant function on G is identified with a function on X± 
invariant under the action 

(x,y) i-> (tx,r 1 y),(x 1 ,x',y 1 ,y / ) i-> (x 1 ,hx',yi, t h~ 1 y'), 

for all t > and /i G GL + (n, R). Then GL + (1, R) x GL + (n, R) is a subgroup 
of G. □ 

It is somewhat mysterious that the left-right action of if on G turns to 
be equivalent to the left action of if x GL + (1, R) C G. 

The next theorem shows the enlargement of the symmetry on invariant 
distributions, which has not been predicted by the geometry, e.g., orbit struc- 
tures. 

Theorem 2 Let n > 3. Then a GL + (n,Ti) -invariant distribution on X\ is 
H-invariant. 

Proof : The invariance is local, so we may consider the restrictions on the 
open subset {Q < 1} and {Q > 0}. (i) On the open subset Q < 1. We can 
take the coordinates (xi, x', y') on the open subset {(x, y) E X\ \ Q < l,x\ ^ 
0} S {(x',y') E R 2n | l x'y' < 1} x { Xl ^ 0}. Actually, y x = (1 - *zy)M- 
The action of if is only on the variables (x',y f ). It has been proved in 
Theorem 1 of [Zj that the D-modules for if -invariants equals to that for H\. 
It means that any if- invariant distribution on this space is if- invariant. On 
the open set y\ ^ 0, we have the same argument. 

(ii) On the open subset Q > 0. We can take the coordinates {(x,y) E 
X 1 \Q>0}3 (x 1 ,x',y 1: y') » ((x',y'/(l- x 1 y 1 )),(x 1 ,y 1 )) E {(£,77) E R 2n | 
*^r] = 1} x {(x\, yi) I Xiyi < 1}. Then both the groups if and if act on the 
first factor transitively. So, the if-invariants implies if -invariants. q 



Remark 3 (1) The proof shows that the same statement for the theorem 
holds if we replace 'distribution' by 'hyperf unction'. (2) We can regards 
Lemma 7.4 of as a special case of theorem; they have proved that an 
H-invariant distribution on X supported on {{x,y) E X± | x' = or y' = 0} 
is supported on {(x,y) E Xi | x' — and y' = 0}. The proof uses the fact 
that distributions are of finite order. 

In the case of the tangent spaces, such an extension of the symmetry has 
been observed in [3] with the eigenfunction condition, and is extended in 
[7] without the eigenfunction condition. These works are inspired by [T2] . 
Compared to the case of the tangent spaces, the geometric setting is slightly 
more subtle for the case of homogeneous spaces. 
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4 Generalized Gelfand pairs 

We define an ant i- involution J : G — » G by J(g) = *g for g £ G. The 
map J induces the linear endomorphism J* on C^ OD (G) HxH by / i— > / o J. 
The natural isomorphism i : G/H\ — > Xi induces the isomorphism : 
c <-oo( Xi ^xgl+(i,r) _^ c -co^hxh % We now describe the map J* on 

Q-oo^HxH ^ C -oo( Xi ^xGL+(l,R)_ We define ^ : jjf x 9 (x U x' ,yi,y') I — ► 

Lemma 4 For any / G C'-^(X 1 )^ xGL+ ( 1 ' R ) ; we /jave JY/ = Tj*/. 
Proof: (i) On Q < 1. Let [/ = {(x,y) G Xi | x x y x > 0}, 

/ ^i -y 2 -y"/yi \ 
a:U x Hi 3 ((x,y),h)\-> x 2 yi G G, 

V^" /„_! / 

where x' = *(x 2 ,a;"), y' = t (y2, t y") and ip : U x H x ^ GL + (n,K) C JT 
by ip((x,y),h) = diag(yi, 1, . . . , l)/y X - The image of a is an open sub- 
set i~ x {{Q > 0}) of G. We have i o J o a = tp ■ (J o i o a). This proves 
(J*i*f)(a((x, y), h)) = f(ioJoa((x, y), h))) = f((f((x, y), h)joioa((x, y), h)) = 
f(j oioa((x,y),h)) = (i*ff)(a((x,y),h)). 

(ii) On Q > 0. Let U = {(x x ,yi) G R 2 | x iyi < 1}, 

(xi -1 + xiyi \ 
1 yi \ti G G, 

J n _! / 

and <p : U x H x H ^ GL + (n, R) C # by y>(/i, /i') = det(hh')- u (hh') G 
GL + (n, R). The image of a is < 1})- We have jo Jo a = (joioa). 

This proves (J*i*f)(oc((x x ,y x ),h,h')) = (i*j*f)(a((xi,yi),h,h')). n 

Theorem 5 For n > 3 ; J* i/ie identity on C~°°(G) HxH . 

Proof. It is enough to prove that j* is the identity on C^ 00 (Xi) HxGL+ ^ 1,It \ 
For any / G C~°°(Xi) H * GL+ ( 1,K \ the support of j*f — / is contained in 
{(x, y) G Xl I a/ = = 0}. Then the distributuion j*/ — / can be written 
as p(n)5(x' ,y') with some polynomial p of the indefinite Laplacian □ = 
Yli=2 d 2 /dxidyi on each open subset {(x,y) G X x \ x x y x > 0, ±x x > 0}. 
This means that (j*f — /) is invariant under the action of j, and that it is 
zero. a 

For the space G/H, the J-criterion that any bi-FFinvariant ezgendistribution 
on G is invariant under J* has been proved in ^Hj- The J-criterion implies 
that the space G/H is a generalized Gelfand pair [12], [T3] . 
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